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This research aimed to evaluate the performance of six ongoing public health 
programs through core performance indicators in Katsina State, Nigeria. 
The healthcare delivery in Africa is mostly program-based. This requires that 
such programs need to be evaluated which may in turn help to identify any 
existing gaps towards the improvement of patients' access and coverage to 
their given service. We identified all active health facilities where our 
programs on malaria, Routine Immunization (RI), Family Planning (FP), 
Tuberculosis and Leprosy (TBL), HIV/AIDS, and Free Medicare (FMC) 
were being carried out. After that, a representative sample was derived to 
obtain data regarding five key performance indicators by using a Logistics 
Indicators Assessment Tool. Of 1,718 facilities, a total of 983 (57.22%) were 
visited, In other words, by assuming a normal distribution; each facility 
expectedly covers only 3,371 individuals. All programs provided different 
and diverse results on each indicator; however, the most obvious challenge 
was in the stock-out and demand vs. receipt of required medications. 
These are particularly for malaria, FMC, FP, and HIV. For instance, 
the stock-out lasted 222 days for malaria and 135 days for FP. Despite this, 
none of the programs had a lower than gold-standard near-term availability 
of required products. Program-based healthcare delivery is inadequate and 
ineffective unless the local system gets simultaneously developed. If required 
medications are not becoming available, optimal access, coverage, 
and benefits cannot be expected to be obtained. Clearly, Nigeria experiences 
a push system of meeting term supplies. Nigeria needs to strengthen its 
pharmaceutical system. 
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1. INTRODUCTION 


Like elsewhere, Africa’s public health is multi-dimensional, multi-actor driven, fragmented, 
and fragile, where everything is likely to be affected by everything else [1]. For instance, in Africa, 
a single sudden change in policy has the capacity to hamper patient’s access to treatment immediately and to 
in turn devoid them of possible benefits in terms of favorable patient-level outcome and overall reduction of 
disease burden in the country [2]. Here, in Africa, the majority of healthcare delivery is “vertical” and 
program-based. Thus, outcomes are dependent on both in-countries as well as external factors [3]. 
Thus, there is a need to systematically examine such public health programs that may help identify 
opportunities, gaps, and challenges that might hamper patient’s access and coverage to any given treatment. 
However, little has been formally done before on this. Assessment of programs is also crucial because 
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elsewhere it is shown that despite enormous program-based fund investment in Africa, little improvement in 
the strengthening of the local healthcare system has been achieved [3]. Thus, continuing with our vision of 
establishing a reliable public health profile and positive international presence of scientifically silent 
locations, and with an objective to systematically assess several ongoing public health programs, 
we performed this work in Katsina (Nigeria). This, if validated, would provide evidence-based conclusions 
on how the programs are functioning in an African cultural context before we may propose improvements, 
for the local and global level. Table 1 gives details of the six (6) supply chain systems 
being investigated [4-10]. 


Table 1. Supply chain systems in Katsina State, Nigeria 











S/N Supply Chain System Number of Health Facilities Number of Local 
Where Services Are Provided Governments Covered 
1 Malaria 1.178 of 1.718 (100%) 34 of 34 (100%) 
2 Family Planning 393 of 1.718 (23%) 34 of 34 (100%) 
3 Routine Immunization 1.623 of 1.718 (94%) 34 of 34 (100%) 
4 Tuberculosis and Leprosy Control 173 of 1.718 (10%) 34 of 34 (100%) 
5 *HIV/AIDS 19 of 1.718 (1.1%) 16 of 34 (47%) 
6 Free Medicare 25 of 1.718 (1.5%) 16 of 34 (47%) 
*HIV/AIDS: Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome; S/N: Serial 
Number 


2. RESEARCH METHOD 
2.1. The significance of disease conditions addressed 

The public health programs that we included were malaria, HIV/AIDS, maternal and child health, 
family planning, routine immunization, tuberculosis, and leprosy. These cover disease conditions of 
preventative, social, and communicable interest; all of which are of particular importance in our country’s 
context [3]. Moreover, these conditions have a cross-therapeutic implication as well. For instance, those with 
cerebral malaria are five times more likely to develop epilepsy and dementia, just like tuberculosis, leprosy, 
and childhood infections for various neurological and mental health conditions. Mainly speaking, maternal 
and child health risk factors may account for 40.0% of epilepsy cases [11-13]. 


2.2. The significance of addressing nigeria 

Nigeria is a large, diverse, and most populous country of Africa. It has about 182 million individuals 
who belong to at least 250 ethnic groups and speaks 350 languages. The population is predominantly youth 
probably because the current life expectancy is merely 55 years. Here, malaria causes 25.0% under-5 year 
deaths, and its caseload may change drastically over a short period time [3]. The HIV burden is second 
highest from among all countries. The incidence of TB is 311/100,000, one of the highest in the World. 
The system-related gaps are also visibly present, for instance, the general workforce for healthcare delivery is 
grossly inadequate. The current public and private expenditure on health (as a percentage of GDP) are 
meager, i.e., 0.8% and 2.6% respectively [3]. There are numerous such examples. 


2.3. Project location 

Katsina is the 5“ most populous and 17" largest province of Nigeria and is located at about 500 KM 
from Abuja, the national capital. It has a population of 5,792,578 (51.0% males, <15 years: 49.0%, >65 
years: 3.0%, life expectancy: 55.2 years according to 2016 census) [14-17]. Most of them belong to Hausa, 
the largest ethnic group of Africa, with contemporary Islam as their predominant religion. Katsina currently 
has a total of 34 Local Government Areas (LGAs), the 2”! most of all in Nigeria. 


2.4. Procedures 

For this work, we prepared a comprehensive list of all active service-delivery facilities that are 
catering to an ongoing public health program (both national and provincial) on malaria (national), 
family planning (national), routine immunization (national), HIV/AIDS (national), tuberculosis and leprosy 
(national), and Free Medicare-women and child (provincial). A simple random sample of facilities was after 
that derived from this source population by assuming that 50.0% facilities would have a "gold-standard 
situation" on five chosen key performance indicators as shown in Table 2, 95% Confidence Interval (CI), 
3.5% variance between groups, and 10.0% total width of CI [18]. 
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Table 1. Sample size determination from number of service points of programs to be visited 








Number of Health Facilities Where Sample 

S/N Program Area Services Are Provided Size 

1 *HIV/AIDS 19 19 

2 Malaria 1.718 315 

3 Tuberculosis & Leprosy Control 173 *DOTS centres 120 

4 Routine Immunization (RI) 1.623 311 

5 Family Planning 393 195 

6 Free Medicare 25 24 





*DOTS: Directly Observed Treatment Short Course; *HIV/AIDS: Human Immunodeficiency 
Virus/Acquired Immune Deficiency Syndrome; S/N: Serial Number 


In order to collect required data, we used a validated questionnaire, named the Logistics Indicator 
Assessment Tool (LIAT) that covered various aspects related to indicators of our interest [19]. 
These included: (a) accuracy of inventory data, (b) demand versus receipt of required health products, 
(c) actual storage conditions in-place, (d) near-term availability of required medications, (e) frequency of 
stock-out of health products. All these indicators were examined in details. For instance, for the accuracy of 
inventory data, the physical balance of the products and most recent requisition reports were matched. 

Similarly, demand versus receipt was examined by referring to the quantities ordered as per 
requisition reports and quantity received as per supplier's proof of delivery. The actual storage conditions 
were examined by using a set of 17 sub-indicators (e.g., the arrangement of products with labels visible, 
physical conditions of cartons and packs, a quarantine area for unusable stock, etc.). Near-term product 
availability was examined by taking stock at-hand vis-à-vis average monthly consumption of the facility to 
see if the stock would last till the next due supply. A stock-out was measured as a point estimate, i.e., 
on the day of the examination, and in the preceding six months, along with the duration of stock-out that may 
have happened. All facilities use an identical number and nature of health products for their part of activities. 
For this work, all eleven (11) field staff recruited received three rounds of training on orientation, practical 
filling of LIAT, and mock data collection. All data were systematically entered in MS-Excel and 
appropriately analyzed. Ethics permission was obtained from the Institutional Review Board of Pôle 
Universitaire Euclide. Informed consent was obtained from all participants before their participation. 
No direct patient-related or any other identification data was collected. 


3. RESULTS AND DISCUSSION 

A total of 1,718 facilities were found to be operational in our province. Of them, a total of 983 
(57.22% of the total) facilities were surveyed across all LGAs in the province. These belonged to HIV/AIDS 
(n=15, 78.9%), Free Medicare-women and children (n=15, 60.0%), tuberculosis and leprosy 
(n=123, 71.1%), family planning (n=195, 49.6%), malaria (n=319, 18.6%), and routine immunization 
(n=311, 19.5%). For convenience, the remaining results are summarized in Tables 3 and Table 4. 


Table 3. Collated data for logistics indicators assessment of six public health programs 
in Katsina State, Nigeria 














Passes Malaria RI FP TBL HIV FMC 
(n=319) (n=316) (n=125) (n=123) (n=15) (n=15) 

Mean staff experience (in months) 181 182 90 71 47 47 
Presence of road 253 215 159 108 15 15 
Presence of electricity 82 74 65 64 15 15 
Presence of water supply 69 74 63 73 15 15 

No. of products used 12 17 9 18 23 165 
Emergency orders placed 50 40 39 16 7 0 
Determine quantities themselves 114 275 118 44 5 0 
6-month stock-out 201 0 99 23 5 15 

Point frequency of stock-out 184 0 75 34 5 15 
No. of days stocked out 222 0 135 1 4 30 
Keep records of expired products 66 37 6 11 1 0 
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Table 4. Rated performance (%) of the core logistics indicator 








Core Logistics Indicator Malaria RI FP TB HIV FMS 
Indicator 1: Accuracy of logistics data for inventory management 98% 99% 91% 96% 97% 100% 
Indicator 2: Percentage difference between the quantity of products 5 6 z 
ordered and the quantity of products received *(the lesser, the better) 100% 1% 8% 2% He 100% 
Indicator 3: Percentage of facilities that maintain acceptable storage 85% 85% 93% 91% 96% 96% 
conditions 
Indicator 4: Percentage of facilities whose stock levels ensure near- 83% 100% 100% 83% 96% 92% 
term product availability 
Indicator 5: Percentage of facilities that experienced a stock out at any 63% 0% 51% 19% 33% 100% 


point during a given period *(the lesser, the better) 





Logistics systems in Nigeria are facing many challenges. The biggest is funding which is somewhat 
unilateral- mostly donors’ fund used to support the entire system [20]. Counterpart funding has been wholly 
ineffective. Consequently, stock-outs loom for many products. Another significant challenge faced by 
logistics systems is the storage conditions of health commodities. Nearly all medical stores are not of 
pharmaceutical grade. Thirdly, data quality has challenges ranging from updating records, missing reports, 
errors in reporting, and quality of data often is inadequate [21]. Logistics technical know-how is also highly 
inadequate among service providers in health facilities. 

The challenges of logistics systems in Nigeria are more pronounced in the Northern part of the 
country where literacy level and coverage are low [22]. The indices of health and logistics of health 
commodities in Katsina State are very poor particularly in the direction of maternal & child health and 
nutrition [23]. Of the five disease areas identified and supported by the Nigeria Supply Chain Integration 
Project (NSCIP), there is no information on the performance of at least the five core logistics indicators [24]. 

We performed a very comprehensive examination of six ongoing public health programs in Katsina 
State, for which we used five key performance indicators. This work was necessary for several reasons but, 
as far as we are aware, this is the first work from Africa that has formally utilized a validated instrument and 
five formal indicators for a program-based evaluation. Earlier efforts have only been restricted to using 
a single indicator, or just being an informal internal evaluation. Our work has, therefore, succeeded to fill 
knowledge-gaps to some extent. 

A total of 1,718 facilities were found to be operational in our province, which implies that 
(assuming a normal distribution) each facility is expected to cover only about 3,371 individuals. This is 
a critical system-related “opportunity” that may help in providing a structured decentralized program-based 
objective-driven health care services for possible sustained performance and patient benefits [25]. In total, 
we surveyed 57.22% of the existing facilities. Thus, our work is reasonably representative of public health 
activities that are taking place, either as a national or a provincial program. For instance, we surveyed 78.9% 
and 71.1% of all facilities where HIV and tuberculosis and leprosy respectively services are provided. 

The choice of our province was also appropriate. For instance, Hausa, the predominant ethnic group 
of Katsina, is 29.0% of the total Nigerian population. Both Katsina and the Nigerian population are similarly 
young (49.0% and 43.0% respectively are <15 years of age). Islam is the predominant religion of Katsina, 
and about 50.0% of Nigerians practice it. There can be numerous such examples. Thus, our results may be 
considered to fairly reflect current public health program-based healthcare delivery situation that is being 
carried out in a Nigerian cultural context, as a whole. 

For any program evaluation and optimal performance, data accuracy is an essential parameter. 
None of our programs had a data deficiency, although numeric differences were found between programs. 
These differences might be due to various factors, such as differences in the “individual capacity” 
(e.g. quality and length of staff experience) and “compensation” (e.g. better dedication towards work), 
or “operational convenience factors” (e.g. distance to home, training facilities), or “donor-related factors” 
(e.g. funding input for better infrastructure), or “political factors” (e.g. centralized or decentralized drug 
supply), etc [26-30]. Elsewhere, data quality issues have been reasonably noted which may be due to 
incessant stock-outs, poor donor engagement, inadequate capacity building of personnel, complexity in 
supply procedures, etc [31-33]. Here, absence of difference in data discrepancy between programs could 
probably be methodological (e.g., inadequate sample size for detecting such a difference). This may also be 
probably because all of our facilities undergo the same kind of staff training and uses the same kind 
(and number) of health products for implementing their part of public health programs. Moreover, our field 
staff was systematically trained for two days, and each indicator was reasonably defined and 
comprehensively evaluated. This may have caused some difference than elsewhere, but this can 
only be speculated. 

The disparity between the quantity of products ordered and those received is a core logistics 
indicator. In the LIAT, this indicator is tracked going by the name, a discrepancy that implies that the lesser 
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the values obtained, the better the performance. Due to the practice of the push system, both malaria and Free 
Medicare programs recorded 100% discrepancy indicating that whatever quantities the facilities requested 
were not the issued amounts to them. For the malaria program, the main reason was due to program 
expansion without corresponding quantification and increased funding while for the Free Medicare, 
it is purely due to insufficient financing which could not afford to buy commodities to satisfy 100% 
of client burden. 

The FP program recorded 8% discrepancies between what they ordered and what they were issued. 
HIV/AIDS, tuberculosis, and routine immunization reported 2%, 2%, and 1% discrepancies. On average, 
the state has 36% discrepancies, but this was caused principally by malaria and Free Medicare programs that 
operate on a push system of issuing commodities to health facilities. Storage conditions form a core indicator 
and are a range of best practices that should be in place to store health commodities. Proper storage 
conditions are undoubtedly required to preserve the potency and efficacy of all products intended for use as 
medicines, diagnostics, or consumables. 

There are many critical things to look at to conclude that storage conditions have been met. In this 
study, up to 17 other indicators have been evaluated to assess this particular indicator. Having adopted 
an 85% recommended pass score for the core indicator; it is evident that all the public health programs 
assessed have passed the assessment. The least scores were 85% scored each by malaria and routine 
immunization programs. Tuberculosis & leprosy, family planning, HIV/AIDS, and Free Medicare are 
performing optimally at 91%, 93%, 96%, and 96% respectively. Overall the state has an average of 91% pass 
for this indicator, and it appears that it is working across the entire state. 

Stock status is a core logistics indicator. All public health programs assessed operate within 
a maximum and minimum inventory control window. This indicator is the flipside of the stock out indicator. 
The percentage pass for stock out having been fixed at not exceeding 20%, a stock status of not less than 
80% is expected to be maintained for a program is considered as passing the indicator. The study found that 
at the time of the assessment, there were high stock levels across all programs. Malaria and 
tuberculosis & leprosy programs had recorded 83% each, Free Medicare program had 92% stock availability, 
HIV/AIDS program maintained stock levels at 96%, and both routine immunization and family planning 
programs maintained 100% stock availability. Overall, public health programs had an average 
stock availability of 92%. 

This indicator is the flipside of stock out rate. In order words, if a stock level of 80% is desired, 
then stock out rate should not exceed 20% .The level of stock availability, the patient load of the different 
logistics system can be a measure of how much commodity security is available for each disease program. 
The routine immunization program had 100% commodity security for 330,353 children that are below the 
age of one year and need to be immunized in the state. 

The family planning program has the second highest patient load with a total of 140,147 clients aged 
between 15—49 years that are using modern contraceptives in 2018. The FP program also had a record of 
100% commodity security for over 140,000 clients accessing services. The next program in terms of the high 
patient load is the Free Medicare program with a record of 65,670 patients in April-June 2018 only. 
As at the time of this assessment, the Free Medicare had 92% availability of commodities. With stock 
availability of 95%, the HIV/AIDS program has 9,435 clients on treatment across 19 health facilities. 
Tuberculosis & leprosy program has 889 adult cases and 524 children cases on treatment, which represent 
a total of 1,413 clients’ burden for the program and had 83% stock availability for all the clients. 
Lastly, the malaria program has the most significant number of facilities supported, but unfortunately, 
there is no established number of people on treatment because malaria is treated per episode of attack. 

Of all indicators we evaluated, the most obvious challenge was in the stock-out, and therefore, 
in demand vs. receipt, of health products as shown in Table 3. This stock-out was visible for all programs 
except routine immunization and tuberculosis and leprosy. This might be interpreted in numerous ways. 
Firstly, nearly all of the facilities in these two programs were estimating their requirement themselves as 
shown in Table 3 which may have helped in part through better assessment of local scenarios. These two 
programs had a relatively low number of stock-out days as well as shown in Table 3. Secondly, the stock-out 
might be in part related to the number of health products being used in a program as shown in Table 3. 
For instance, Free Medicare uses 165 products, and 62.0% of its facilities were out of stock for six months as 
shown in Table 3. Thirdly, stock-out in Africa should not be unexpected because more than 90.0% of health 
products in Africa are imported from outside [34-40]. This, therefore, is subjected to external factors as well 
as in-country regulations and restrictions. To elaborate further, Nigeria has about 130 pharmaceutical 
companies, but only 8.0% of them have a sustainable operation. A formal evaluation of them has shown poor 
performance, for instance, capacity utilization of just about 40.0% [41]. Having no local availability of active 
ingredients is another significant hindrance towards a desirable level of availability and access to essential 
medications. No doubt, the pharmaceutical sector here accounts for merely 0.2% of Nigeria’s GDP [42]. 
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While comparing point (on the day of observation) and six-monthly stock-out, Table 3, there was 
not much subsequent decline in the stock-out. For instance, in malaria and family planning, the stock-out was 
for 222 and 135 days, respectively, Table 3. Thus, poor pharmaceutical scenario (e.g., stagnant supply, 
over-reliance on external stakeholders, in-country challenges) is a significant feature of the healthcare 
delivery system here in Nigeria. This again should not be unexpected since elsewhere it is shown that 
donor-funded programs do not lead to simultaneous broad “horizontal” development of local healthcare 
delivery system [42]. For instance, a centralized procurement does not always mean a timely transport of 
supplies to provinces. Fourthly, another reason of stock-out might be an abrupt change in caseload as it may 
be seen in emergency orders placed by the programs, Table 3, such as malaria. This should not be unexpected 
in population like ours where caseload may change by 52.0% over a relatively short period [3]. 
Moreover, in Nigeria, disease conditions such as malaria are generally treated per episode of attack and not 
by its total number of caseload. 

Lastly, as far as we are aware, ours was the first comprehensive attempt to evaluate, and in turn 
identify challenges for, the delivery of public health programs in Africa. Our work visibly has its application 
across various interest areas including health policy making, critical decision and risk analysis, access and 
delivery of quality healthcare, health system evaluation, and is likely to strengthen our knowledge from 
a systems perspective; wherein everything is likely to be affected by everything else. Nevertheless, our work 
does not include those programs that are periodic or time-bound. Our work was not designed to identify 
predictors that may have affected performance between different programs and facilities. For instance, 
the military rule is common in Nigeria (until 1999 when democracy came to stay) that may affect the 
performance, benefit, and continuity of programs. 


4. CONCLUSION 

Positively, six sizeable public health programs were overall found to meet most of the key 
performance indicators in a systematic evaluation of 983 health facilities. Negatively, most programs 
(except routine immunization and tuberculosis & leprosy) fail in the parameter of stock-out of required health 
products. Our work also implies that program-based healthcare is of little long-term benefit for installing 
a “broad” improvement in the healthcare delivery system in the host country. Our work also implies that 
public health programs in Nigeria quite likely to follow a “push” system to meet their essential supplies. 
Lastly, our work calls policymakers to adopt a self-help model through strengthening the local 
pharmaceutical sector in order to bring better patient benefits and meet relevant public health and 
development goals. 
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